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FIG. 1 

PRIOR ART 

1 c:\collections 

2 notes.txt 

3 myletter.doc 

4 c-myhomepage 
5 

6 s 
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8 myphoto.jpg 
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1 c:\coIlections 

2 notes.txt 
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FIG. 3 

102 

y 

1 collection c-myhomepage 

2 coil-type cf-web-page 

3 coll-desc A sample homepage collection 

4 end-collection 
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FIG. 8 

1 /* collection data structure */ 

2 collection-info { 

3 + specifierjnfo 

4 + coll-type-indicator 

5 + other specifier information . . . 

6 + content J nfo 

7 + content Jocation jnfo . . . 

8 + content jmembers . . . 

9 + other content information... 

1 0 + other collection structure information. . . 

11 } 



FIG. 9 

1 /* collection type definition data structure */ 

2 collection-type-definition-info { 

3 + coil-type-name 

4 + collection internal structure info ... 

5 + collection content location info ... 

6 + collection content type recognition info . . . 

7 + other collection type definition information. . . 

8 } 
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FIG. 10 
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1 /* collection type internal structure definitions */ 

2 dir_source_files As 

3 dir_doc_files Adoc 

4 /* content location definitions (per-type content links) 7 

5 content_subtree_http http://host.com/some/dir/name 

6 content_subtree_ftp ftp://host.com/some/dir/name 

7 content_subtree_nfs /some/local/directory/name 

8 /* content type recognition definitions 7 
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content 
content 
content 
content 
content 
content 
content 



^policy subtree_below_cspec_file 

_file_type .c file_cpp 

Jilejype ,c file_c 

_file_type .h file_c_include 

_file_type .doc file_ms_word 

Jilejype .html file_html 

Jile_type .xls file_ms_excei 



16 /* collection processing definitions 7 



17 compile_cJiles 

18 compiler_windows 

19 compiler_unix 

20 build platforms 

21 process files 

22 link libraries 



yes 

vc++ 

gcc 

Win98, Win2000, linux 
compile link 
stdio math sock 



23 /* results dispatching definitions 7 

24 results_ftpJiost ftp.output.com 

25 resultsjftp_dir c:\ftphome\collection\results 
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FIG. 12 

1 /* simplified cca algorithm */ 

2 Call get runtime information to obtain a list of 
commands to apply 

3 Call collection list producing means to obtain a 
list of target collections for command application 

4 Call command execution means to apply commands 
to target collections 



FIG. 13 

1 /* runtime information data structure */ 

2 runtime-info { 

3 + invocation options 

4 + runtime environment information 

5 + commands to execute on recognized collections 

6 + collection recognition criteria 



7 

8 } 
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FIG. 14 
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FIG. 15 



1 /* simplified algorithm for list producing means */ 

2 Build data structures 

3 /* Obtain list of target collections for command application */ 

4 - add explict invocation-provided collections to list 

5 - add collection-recognizer-provided collections to list 

6 - add collections from other sources to list 

7 /* Build sorted-colls data structures */ 

8 - create list of target collections sorted alphabetically 

9 - create list of target collections sorted by visit order 

1 0 - create list of target collections sorted by other means 

1 1 Return completed coll-list-info data structure to caller 
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FIG. 16 

1 /* list of target collections for command application */ 

2 target-coll-list { 

3 + list of collection-structures (eg FIG 8) 

4 } 



FIG. 17 

1 /* structure for holding sorted lists of collections */ 

2 sorted-colls { 

3 + sort-type = ALPHA, VISIT_ORDER, . . . 

4 + sorted-collection-set { 

5 + coll-1 in sort order 

6 + coll-2 in sort order 

7 + ... 

8 } 



FIG. 18 

1 /* structure for holding collection list info */ 

2 coll-list-prod-info { 

3 + list of collection structures (target-coll-list) 

4 + list of collection type definition structures (FIG 9) 

5 + list of sorted-colls structures 

6 + collection recognition info ... 



7 



} 
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FIG. 19 

1 c:\coIlections 

2 programs 

3 helloworld 

4 c-hello-library 

5 c-hello 

6 c-myprogram 

7 parts 

8 c-include-files 

9 c-library-one 

10 c-Iibrary-two 

1 1 webstuff 

12 c-myhomepage 

1 3 c-myphotos 



FIG. 20 

0 /* File colls-fig-20.txt holds list of collections for tree FIG 1 9 7 

1 c:\collections\programs\helloworld\c-hello-library 

2 c:\coliections\programs\helloworld\c-hello 

3 c:\collections\c-myprogram 

4 c:\collections\parts\c-include-files 

5 c:\collections\parts\c-library-one 

6 c:\collections\parts\c-library-two 

7 c:\collections\webstuff\c-myhomepage 

8 c:\collections\c-myphotos 
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FIG. 21 
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FIG. 22 

1 /* simplified algorithm for execute sequential direct means */ 

2 Build data structures 

3 /* walk list of target collections */ 

4 For each coll in list of one-coll-cmd-exe structures 

5 - change working directory to desired execution directory 

6 /* execute direct commands on each target collection */ 

7 - for each command in list of cmd-exe-status structures 

8 - execute the command using a subordinate helper module 

9 - record command status and errors 

10 - if command status = FAILS continue with next collection 

1 1 - continue with next command in list of commands 

12 /* clean up and return status */ 

1 3 Change working directory back to original invocation directory 

14 Return overall execution status to caller 
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FIG. 23 

1 /* command and status info for 1 command */ 

2 cmd-exe-status { 

3 + command to execute 

4 + return code status of execution attempt 

5 + other error information 

6 + ... 

7 } 

FIG. 24 

1 /* N commands and status info for 1 target collection */ 

2 one-coll-cmd-exe { 

3 + target collection for commands 

4 + list of cmd-exe-status structures 

5 + ... 

6 } 



FIG. 25 

1 I* commands and status info for all target collections 7 

2 all-coll-cmd-exe { 

3 + list of coli-cmd-exe structures 

4 + execution-type = DIRECT, INDIRECT 

5 + exe-direct-method = FORK, THREAD, . . . 

6 + exe-indirect-method = BATCH, PERL, . . . 



7 } 
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FIG. 26 

1 if not ()==(% 1) goto goodargs 

2 echo Usage: doinseqcommandl c2 c3 ... c5 

3 echo doinseq make all 

4 echo doinseq dir*c 

5 goto quit 

6 :goodargs 

7 cd c:\coliections\c-myphotos\win98.plt 

8 call %1 %2 %3 %4 %5 %6 %7 %8 %9 

9 cd c:\collections 

10 cd c:\collections\parts\c-library-one\win98.plt 

1 1 call %1 %2 %3 %4 %5 %6 %7 %8 %9 

12 cd c:\collections 

13 cd c:\collections\programs\heIloworld\c-helIo\win98.plt 

14 call %1 %2 %3 %4 %5 %6 %7 %8 %9 

15 cd c:\collections 

16 ... 

17 :quit 



FIG. 27 

1 C:\> cca doinseq -explicit-colls colls-fig-20.txt -platform win98.plt 

2 C:\> doinseq <a-command-to-execute> 

3 C:\> doinseq Is 

4 C:\> doinseq make all 
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FIG. 28 

1 /* simplified algorithm for execute sequential indirect */ 

2 Build data structures 

3 Generate a batch file template framework to hold commands 

4 /* walk list of collections and generate commands into the batch file */ 

5 For each coll in list of coll-cmd-exe structures 

6 - emit command to change working directory into 

desired execution directory 

7 - emit script file argument variables to hold commands that 

are passed in to the script file for execution 

8 - emit command to change back to original working directory 

9 - continue with next collection in list of collections 

10 Return batch file execution status 
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FIG. 29 

1 /* rec-coll recognized-collections data structure */ 

2 rec-coll { 

3 + rec-coll-list 



4 + coll-structure-1 

5 + cspecjnfo ... 

6 + ctypejdefjnfo ... 

7 + eContent jnfo ... 

8 + other_colljnfo 

9 + coII-structure-2 

10 + cspecjnfo ... 

11 + ctype_defjnfo ... 

12 + eContent jnfo ... 

13 + other_coll_info 

14 + coll-structure-3 
15 



16 + other collection recognition info 

17 } 
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FIG. 30 

1 c:\coilections 

2 programs 

3 helloworld 

4 c-hello ctype = cf-program 

default vo=100 

5 c-helio-library ctype = cf-library 

default vo=50 

6 c-myprogram ctype = cf-program 

default vo=100 

7 parts 

8 c-include-fiies ctype = cf-includes 

default vo=10 

9 c-library-one ctype = cf-library 

default vo=10 

1 0 c-library-two ctype = cf-library 

EXPLICIT vo=49 

1 1 webstuff 

1 2 c-myhomepage ctype = cf-doc-html 

default vo=100 



13 



c-myphotos 



ctype = cf-web-page 
default vo=100 
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FIG. 31 
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FIG. 32 

1 collection c-Iibrary-two 

2 colljype cf-library 

3 coll_desc A library with explicit visit order 

4 coll-visit-order 49 

5 end-collection 
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FIG. 34 

1 /* simplified visit order algorithm */ 

2 Receive unsorted list of collections 

3 Obtain numeric visit order values for each collection in list 

4 Sort the list of collections according to execution visit order 

5 Write sorted information to sorted-colls data structure (FIG 17) 

6 Return sorted-colls data structure to calling module 



FIG. 35 
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FIG. 36 



1 if not ()==(% 1 ) goto goodargs 

2 echo Usage: doinseq commandl c2 c3 ... c5 

3 echo doinseq make all 

4 echo doinseq dir*.c 

5 goto quit 

6 :goodargs 

7 cd c:\collections\parts\c-include-files\win98.plt (vo=10) 

8 call %1 %2 %3 %4 %5 %6 %7 %8 %9 

9 cd c:\collections 

10 cd c:\collections\parts\c-library-two\win98.plt (vo=49) 

1 1 call %1 %2 %3 %4 %5 %6 %7 %8 %9 

12 cd c:\collections 

13 cd c:\collections\parts\c-library-one\win98.plt (vo=50) 

14 call %1 %2 %3 %4 %5 %6 %7 %8 %9 

1 5 cd c:\collections 

1 6 cd c:\collections\programs\helloworld\c-hello-library\win98.plt 

1 7 call %1 %2 %3 %4 %5 %6 %7 %8 %9 (vo-50) 

18 cd c:\collections 

19 cd c:\collections\programs\helloworld\c-hello\win98.plt 

20 call %1 %2 %3 %4 %5 %6 %7 %8 %9 (vo=1 00) 

21 cd c:\collections 

22 ... 



23 :quit 
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FIG. 39 

0 vo-doc.def: 

1 cf-doc-sgml 10 

2 cf-doc-html 10 

3 cf-doc-indexes 20 
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FIG. 40 
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1 /* simplified algorithm for calculating parallel execution groups */ 

2 Obtain list of target collections, sorted into proper visit order 

3 Obtain physical parallelism limit = phys_par_limit 

4 Obtain administrative parallelism limit = adminjimit 

5 /* calculate problem parallelism limit */ 

6 set problem jaarjimit = 1 

7 For each unique visit order in list of target collections, 

8 - count number of collections with current visit order value 

9 - if current_count > problem jrarjimit, 

1 0 set problem_par_limit = current_count 

1 1 /* calc min of problem, physical, and admin parallelism limits */ 

12 usefuLparJimit = min (problemjDarJimit, physical jparjimit, 

admin_parjimit) 

13 /* calc parallel execution groups using usefuLparJimit */ 

1 4 For each unique visit order in list of target collections, 

15 - create a new parallel execution group 

1 6 containing all collections that match the current visit order 

17 - if group size is > usefuLparJimit, 

1 8 split group into smaller groups 

1 9 until no groups exceed usefuLparJimit 

20 - continue with next unique visit order in list of target collections 

21 Return parallel execution ordering and limits to caller 
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FIG. 43 



1 /* simplified algorithm for execute parallel direct */ 

2 Build data structures 

3 /* calculate parallel execution groups */ 

4 Calculate parallel execution groups per algorithm FIG 42 

5 /* execute collections in each parallel group in parallel */ 

6 For each parallel execution group, in proper parallel exe order { 

7 - Execute in parallel { 

8 - For each collection in the parallel exe group 

9 - change working directory to desired execution directory 

10 /* execute commands for one collection */ 

11 - for each command in list of cmd-exe-status structures 

1 2 - execute the command 

1 3 - record command status and errors 

14 - if command status = FAILS continue with next collection 

1 5 - continue with next command in list of commands 

16 - Wait for all collections in the group to finish 

17 } /* end of parallel section beginning on Line 7 

18 } /* end of parallel execution group list traversal beginning on Line 6 

19 Return overall execution status information to caller 
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/* collections for 1 parallel execution group 



*/ 



parallel-exe-group { 



+ par-exe-rank = 1, 2, 3, ... 
+ par-exe-group-coil-list { 



+ collection-1 



+ collection-2 



+ 



} 
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FIG. 45 

/* collections for all parallel execution groups */ 
cmd-exe-parallel { 

+ list of coll-cmd-exe structures 

+ list of parallel-exe-group structures 

+ exe-type = DIRECT, INDIRECT 

+ exe-method-direct = FORK, THREAD, ... 

+ exe-method-indirect = PERL, PROGRAM, 

} 



FIG. 46 

1 time=0, vo=10 c-include-files 

2 time=1, vo-49 c-library-two 

3 time=2, vo=50 c-library-one 

4 c-hello-Iibrary 

5 time=3, vo-100 c-hello 

6 c-myprogram 

7 c-myhomepage 

8 c-myphotos 
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FIG. 47 

1 /* simplified algorithm for execute parallel indirect */ 

2 Build data structures 

3 /* calculate parallel execution groups */ 

4 Calculate parallel execution groups per algorithm FIG 42 

5 /* execute collections in each parallel group in parallel */ 

6 For each parallel execution group, in proper parallel exe order { 



7 - Emit commands for executing in parallel { 

8 - For each collection in the parallel exe group 

9 - emit cd command to change to desired execution directory 

1 0 /* emit execution commands for one collection */ 

11 - for each command in list of cmd-exe-status structures 

12 - emit syntax for the command to be executed in parallel 

1 3 - emit syntax to record command status and errors 

14 - continue with next command in list of commands 

1 5 - Emit syntax to wait for all collections in the group to finish 

16 } /* end of parallel section beginning on Line 7 

17 } /* end of parallel execution group list traversal beginning on Line 6 

18 Return overall execution status information to caller 
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#!/bin/sh 



if [ $# -It 1 ] ; then 

echo "Usage: doinparallel command-1 c-2 c-3 ... c-N' 
echo " doinparallel copy filel file2" 

echo " doinparallel make all" 

exit 1 



fi 



8 # at time 0, apply parallel execution group #1 

9 cd /collections/parts/c-include-files/linux.plt 

10 $@ 

11 cd /collections 

12 ... # execution group #2 omitted to save space 

13 # at time 2, apply commands parallel execution group #3 

14 cd /collections/parts/c-Iibrary-one/linux.plt 

15 $@& 

16 cd /collections/programs/helloworld/c-hello-library/linux.plt 

17 $@& 

1 8 # wait for all parallel jobs to complete 

19 wait 

20 cd /collections 

21 # at time 3, apply commands to parallel execution group #4 

22 cd /collections/programs/helloworld/c-hello/linux.plt 

23 $@& 

24 cd /collections/c-myprogram/linux.plt 

25 $@& 

26 cd /collections/webstuff/c-myhomepage/linux.plt 

27 $@& 

28 cd /collections/c-myphotos/linux.plt 

29 $@& 

30 # wait for all parallel jobs to complete 

31 wait 

32 cd /collections 

33 exitO 
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FIG. 49 

1 c:\colIections 



2 programs 

3 helioworld 

4 c-hello 

5 cspec 

6 s... 

7 win98.plt... 

8 gnulinux.plt... 

9 c-myprogram 

10 cspec 

11 s... 

12 win98.plt... 

13 gnulinux.plt... 

14 parts 

15 c-include-files 

16 cspec 

17 s... 

18 win98.plt... 

19 gnulinux.plt... 
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1 Visit all s dirs beside cs| 

2 Visit all immediate child 

3 Format files in all s dirs 

4 Modify all cspec files 

5 Delete all collections 

6 Clean up win98.plt dirs 

7 Delete all pit dirs 



ec files, non-recursively 
dirs, recursively 
(visit s dirs) 

(visit root dirs of collections) 
(visit parent dirs of collections) 
(visit win98.plt dirs) 
(visit root dirs of collections) 



